Skip to content

fix(trace stats): Configure span kind and peer tag for agent-side stats#1189

Open
lucaspimentel wants to merge 12 commits into
mainfrom
lpimentel/fix-stats-span-kind-peer-tags
Open

fix(trace stats): Configure span kind and peer tag for agent-side stats#1189
lucaspimentel wants to merge 12 commits into
mainfrom
lpimentel/fix-stats-span-kind-peer-tags

Conversation

@lucaspimentel

@lucaspimentel lucaspimentel commented Apr 16, 2026

Copy link
Copy Markdown
Member

Overview

Configure SpanConcentrator in bottlecap with the Go agent's default ComputeStatsBySpanKind span kinds and basePeerTags peer tag keys, fixing two gaps in agent-side trace stats computation:

  • APMSVLS-464: span_kinds_stats_computed was empty, so non-top-level, non-measured spans with span.kind = server/client/producer/consumer) were silently excluded from stats.
  • APMSVLS-463: peer_tag_keys was empty, so client/producer/consumer spans had no per-dependency granularity in stats output (e.g., all S3 buckets, DynamoDB tables, Kafka topics lumped together).

Changes

  • Add STATS_ELIGIBLE_SPAN_KINDS constant (4 span kinds matching the Go agent's KindsComputed)
  • Add DEFAULT_PEER_TAG_KEYS constant (43 peer tag keys matching the Go agent's basePeerTags from mappings.json)
  • Pass both to SpanConcentrator::new() instead of empty vecs
  • Add unit tests that exercise the StatsConcentratorService pipeline for both features
  • Add fake-intake integration tests that route real spans through the concentrator → flusher → intake path, verifying span kind and peer tags survive msgpack/gzip serialization end to end

Follow-up

Both constants are hand-copied from the Go agent (KindsComputed and basePeerTags), which is the source of truth, and the same lists are also duplicated in serverless-components. A future refactor should keep the Rust agents in sync with the Go agent rather than each maintaining its own copy.

Testing

  • test_span_kind_stats_computed: sends a non-root, non-measured client span through the service, verifies stats are produced with span_kind="client"
  • test_peer_tags_populated: sends a client span with db.instance and db.system meta, verifies peer_tags contains both in the stats output
  • stats_span_kind_through_fake_intake: feeds a non-root, non-measured server span through the full concentrator → flusher → fake-intake path, asserts the decoded StatsPayload carries span_kind="server"
  • stats_peer_tags_through_fake_intake: feeds a client span with db.instance/db.system meta through the same path, asserts the decoded peer_tags contain both keys after serialization
  • Tests were written first and confirmed failing before the fix was applied
  • Full test suite still passes

"Turns out an empty vec is a great way to compute stats on absolutely nothing. Efficient, but not very useful." — Claude 🤖

@lucaspimentel lucaspimentel changed the title [APMSVLS-464][APMSVLS-463] fix: Configure span kind and peer tag stats in bottlecap [APMSVLS-464][APMSVLS-463] fix: Configure span kind and peer tag stats Apr 16, 2026
@lucaspimentel lucaspimentel force-pushed the lpimentel/fix-stats-span-kind-peer-tags branch from c06bc9b to 2eb4403 Compare June 10, 2026 17:58
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 10, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 4 Pipeline jobs failed

Bottlecap (Rust) | Test Suite   View in Datadog   GitHub Actions

DataDog/datadog-lambda-extension | bottlecap (amd64)   View in Datadog   GitLab

DataDog/datadog-lambda-extension | bottlecap (amd64, alpine)   View in Datadog   GitLab

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0584c5a | Docs | Datadog PR Page | Give us feedback!

@lucaspimentel lucaspimentel marked this pull request as ready for review June 10, 2026 21:49
@lucaspimentel lucaspimentel requested review from a team as code owners June 10, 2026 21:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR configures Bottlecap’s SpanConcentrator to match the Go agent defaults for (1) which span.kind values are eligible for stats computation and (2) which peer-related tag keys are used to populate per-dependency peer_tags, closing gaps in agent-side trace stats aggregation.

Changes:

  • Configure SpanConcentrator with the Go agent’s default computed span kinds (client, server, producer, consumer) instead of an empty list.
  • Configure SpanConcentrator with the Go agent’s default peer tag key set (basePeerTags) instead of an empty list.
  • Add end-to-end tests for span-kind-based stats eligibility and peer tag population in emitted stats.

Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
Comment thread bottlecap/src/traces/stats_concentrator_service.rs
Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
Comment thread bottlecap/src/traces/stats_concentrator_service.rs Outdated
@lucaspimentel lucaspimentel force-pushed the lpimentel/fix-stats-span-kind-peer-tags branch from 4ce8868 to 5e31b4f Compare June 11, 2026 22:34
@lucaspimentel lucaspimentel changed the title [APMSVLS-464][APMSVLS-463] fix: Configure span kind and peer tag stats fix: Configure agent-side span kind and peer tag stats Jun 11, 2026
@lucaspimentel lucaspimentel changed the title fix: Configure agent-side span kind and peer tag stats fix: Configure span kind and peer tag for agent-side stats Jun 11, 2026
Tests verify that StatsConcentratorService computes stats
for spans with span.kind and populates peer_tags in output.
Both tests currently fail due to empty span_kinds_stats_computed
and peer_tag_keys vecs passed to SpanConcentrator::new().

🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
Pass STATS_ELIGIBLE_SPAN_KINDS and DEFAULT_PEER_TAG_KEYS to
SpanConcentrator::new() to match the Go agent defaults. This
enables stats for OTel spans with span.kind and adds per-dependency
granularity via peer tags for client/producer/consumer spans.

🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
Cover the concentrator's span-kind and peer-tag eligibility through the
full concentrator -> flusher -> intake path, verifying both survive
msgpack/gzip serialization.
@lucaspimentel lucaspimentel force-pushed the lpimentel/fix-stats-span-kind-peer-tags branch from 5e31b4f to a652afd Compare June 12, 2026 21:48
The enrich_ctx_keeps_tracer_set_cold_start_trace_id_without_tracer_detected
test calls setup(), which spawns a Tokio task, but was annotated with #[test]
instead of #[tokio::test], causing a "no reactor running" panic. Match its
sibling test and use #[tokio::test] async fn.

🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
@lucaspimentel lucaspimentel changed the title fix: Configure span kind and peer tag for agent-side stats fix(trace stats): Configure span kind and peer tag for agent-side stats Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants